Recover from non-monotonic presentation timestamps in MP4 video tracks without ctts - #3352

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Pushed a fix for a second bug: audio/video would slowly drift out of
sync over long playback. Cause was an accumulating rounding error in
the per-frame timestamp math — invisible on a 90s test clip, ~58ms by
the end of a 2-hour movie. Fixed by rounding only the final offset,
not the per-frame duration.

Tested in two real players on the affected hardware (Jellyfin Android
TV, and Plezy). 40 minutes of real playback, no drift, no stutter.

Thanks for the detailed investigation and for providing a legal,
reproducible sample. I agree that changing the late-frame threshold
does not address the underlying problem.

However, I don't think the current evidence isolates this as a
MediaTek decoder/HAL defect yet. The repro MP4 itself appears to have
a questionable presentation timeline for reordered frames.

I downloaded tos_x265_3.3_final.mp4 and inspected it with ffprobe.
All 360 video packets have PTS == DTS, even though the HEVC stream
uses B-frames and B-pyramid. For example:

packet PTS/DTS:
-0.080000 / -0.080000
-0.041667 / -0.041667
 0.000000 /  0.000000
 0.041667 /  0.041667
...

After decoding, the propagated frame PTS becomes non-monotonic:

frame PTS:
0.041667
0.083333
0.000000
0.125000
0.250000
0.291667
0.208333
...

while FFmpeg's best_effort_timestamp reconstructs approximately:

0.041667
0.083333
0.125000
0.166667
0.250000
0.291667
0.333333
...

This looks consistent with missing or incorrect composition offsets
in the MP4, possibly introduced by the raw-HEVC stream-copy muxing
pipeline in the repro README.

Android documents BufferInfo.presentationTimeUs as being derived
from the timestamp queued with the corresponding input buffer:

https://developer.android.com/reference/android/media/MediaCodec.BufferInfo#presentationTimeUs

Therefore, reproducing the same output timestamp sequence with
AMediaCodec/AMediaExtractor proves that ExoPlayer did not introduce
the sequence, but it does not by itself prove that the decoder/HAL is
behaving incorrectly. It may simply be propagating input timestamps
that do not describe the HEVC presentation order correctly.

Could we rule this out before changing MediaCodecRenderer?

1. Generate the same encoded content in a container with correct
   composition timestamps and verify that some packet PTS values
   differ from DTS where reordering requires it.
2. Log each timestamp queued into MediaCodec and its corresponding
   output BufferInfo.presentationTimeUs.
3. Compare packet PTS/DTS for the affected x265 3.3 file and the
   unaffected x265 4.1 file.

There is also a separate scope concern with the current patch: despite
being motivated by one MTK decoder, it rewrites timestamps for every
video decoder whenever Format.frameRate is known. This would replace
valid VFR timing, intentional gaps, duplicate frames, and format
transitions with a synthetic CFR timeline. The frame duration is also
not reset when the input frame rate changes.

The patch explains why this particular CFR sample becomes smooth, but
I think the source timeline needs to be ruled out before treating it
as a general Media3 renderer fix. If a player-side workaround is still
needed afterward, it should probably be gated to the affected codec
and device, activated only after detecting the known timestamp pattern,
and kept out of the generic MediaCodecRenderer path.

Summary

MANY thanks for your helpful reply. We looked into all three of your suggestions. Short version: I no longer think this is a
MediaTek decoder defect. It's a bug in the reproduction repo's own muxing pipeline (which reflects bugs in the muxing pipelines of a bunch of files running around,) and as far as
we can tell it's independent of x265 version, encoder options (beyond whether B-frames are used
at all), and source content.

1. Composition timestamps: correctly-muxed vs. naively-muxed content

You were right about the repro file. Reproduced your ffprobe findings exactly on
tos_x265_3.3_final.mp4: every packet has PTS == DTS (0/360), and the propagated frame PTS is
non-monotonic in the pattern you posted, while best_effort_timestamp reconstructs a clean
sequence.

The repro README's mux step (ffmpeg -fflags +genpts -r 24 -i raw.hevc ... -c:v copy)
stream-copies a detached raw HEVC stream, which carries no composition-time
information of its own. +genpts fills the gap by assigning PTS = DTS = packet_index / fps —
decode order — which is only correct when there's no B-frame reordering to begin with. Minimal
confirmation, x265 4.1, defaults, nothing else changed:

ffmpeg -f lavfi -i "testsrc2=size=640x360:rate=24:duration=15" -f yuv4mpegpipe src.y4m

# encode straight to MP4 — libavformat assigns real composition offsets in-process
ffmpeg -i src.y4m -c:v libx265 -tag:v hvc1 good.mp4

# encode to a detached raw stream, then remux exactly like the repro README does
ffmpeg -i src.y4m -c:v libx265 -f hevc raw.hevc
ffmpeg -fflags +genpts -r 24 -i raw.hevc -c:v copy -tag:v hvc1 bad.mp4

good.mp4:  PTS != DTS on 270/360 packets  (correctly timed)
bad.mp4:   PTS != DTS on   0/360 packets  (every packet's PTS is just its DTS)

Same encoder, same options, same source — only the mux step differs.

Confirmed on-device too, on a build with none of our patches applied: correctly-muxed B-frame
content plays clean; content muxed the repro's way drops ~30% of frames, consistently, across
every file we've tried.

2. x265 3.3 vs. current x265 4.1

Re-encoded the same Tears of Steel source with current x265 (4.1+1-1d117be), options matched as
closely as possible, muxed with the unmodified repro pipeline. Result: indistinguishable from
the 3.3 encode — 0/360 PTS != DTS, same broken pattern.

This briefly looked like it contradicted our own earlier finding that x265 4.1 content didn't
reproduce the bug, so we ran it down rather than gloss over it. The earlier "unaffected" 4.1 file
has real composition offsets (PTS != DTS on 243/360 packets) — it was muxed correctly, almost
certainly via a single-step ffmpeg -c:v libx265 ... output.mp4 encode, not the raw-stream-then-
remux process the repro pipeline uses. That two-step process isn't optional for x265 3.3: its CLI
has no muxer at all (-o is raw-bitstream-only). Confirmed both ends on-device: the original file
still plays with zero dropped frames; a freshly built, zero-options x265 4.1 file put through the
naive remux drops 682/2280 frames (29.9%). Same decoder, same device, same encoder version — only
the mux method differs.

So: not a contradiction, and not an x265-version-specific bug. Encoder version is irrelevant to
the trigger; whether the file went through a proper muxer is the entire story.

3. Logging queued input vs. dequeued output timestamps

Patch (diagnostic-only, no behavior change):
6fc70188e49667a894b44ada761193e28ebda8b9
on our fork, based on the clean release branch (i.e. without our reorder/relabel patch from this
PR applied — this measures the raw, unpatched decoder). Logs the timestamp on every
queueInputBuffer/queueSecureInputBuffer call and every buffer dequeued from
drainOutputBuffer. Built, installed on the real device, captured against
tos_x265_3.3_final.mp4 and the 4.1 re-encode:

Queued-input order is perfectly monotonic in both — a direct echo of the container's decode-order
timestamps. Dequeued-output order is non-monotonic in both, at essentially the rate the container
alone predicted, with matching non-monotonic transition offsets between the two files.

The key check: every single dequeued output timestamp, in both files, is one of the exact
values that was queued as input — zero exceptions. c2.mtk.hevc.decoder never emits a
timestamp it wasn't given. Direct on-device confirmation of the presentationTimeUs contract you
cited — the decoder echoes back exactly what it was handed; the non-monotonicity is already
present in what it was handed.

Where this leaves things

I don't think this is evidence of a MediaTek decoder defect. However, what it does look like is
that files encoded with the x265 3.x series are likely to actually hit this in practice: its CLI
has no muxer of its own, so any pipeline built around it needs a separate remux step, and if that
step doesn't compute real composition offsets — ours doesn't, and we don't know what tooling
popularly shared media pipelines typically used either — this is exactly the result.

VLC plays these incorrectly-timestamped files just fine, so I want to dig into why. My guess is
it comes down to VLC's "if it's only a little late, just display it right now" approach versus
shouldDropOutputBuffer, which is closer to "if the presentation timestamp says it's late, chuck
that frame right in the garbage." I'll look into how VLC actually handles this (and crucially, how it is not overly permissive, as you have cautioned against) and follow up.

Why VLC doesn't stutter on this content

Instrumented VLC itself (not just read its source) and ran it on-device against the same repro file. It receives equally or more disordered decoder output than ExoPlayer from c2.mtk.hevc.decoder — 52.2% non-monotonic steps in the raw echoed presentationTimeUs, vs. ~42% measured earlier from ExoPlayer, same 1×–4× frame-period jump signature. So it's not decoder behavior, and it's not a threading/queuing difference between the two clients.

VLC just never lets that value reach anything that schedules a frame. Its MP4 demuxer refuses to invent a PTS for a video track when the file has no ctts box — true of any file built via the raw-ES + -fflags +genpts pipeline this whole thread has been using:

// mp4.c, https://code.videolan.org/videolan/vlc/-/blob/ac6c2a405d652b5576128ceb9fec2c342f0e83ec/modules/demux/mp4/mp4.c#L1391-L1396
if( MP4_TrackGetPTSDelta( p_demux, tk, &i_delta ) )
    p_block->i_pts = p_block->i_dts + i_delta;
else if( tk->fmt.i_cat != VIDEO_ES )
    p_block->i_pts = p_block->i_dts;
else
    p_block->i_pts = VLC_TICK_INVALID;   // video track, no ctts: don't guess

That VLC_TICK_INVALID trips a fallback in the MediaCodec wrapper that relabels every output picture by decode-order index instead of trusting the (disordered) decoder echo — confirmed on-device: 0/2280 non-monotonic after the fallback, vs. the 52.2% it replaced.

androidx/media3's MP4 extractor has no "PTS unknown" state. When ctts is absent it silently produces PTS = DTS, because the offset term it would add just stays at its zero default:

// BoxParser.java, https://github.com/androidx/media/blob/5fb306449733dd71595700c1227ad6087578c559/libraries/extractor/src/main/java/androidx/media3/extractor/mp4/BoxParser.java#L516
int timestampOffset = 0;
...
// https://github.com/androidx/media/blob/5fb306449733dd71595700c1227ad6087578c559/libraries/extractor/src/main/java/androidx/media3/extractor/mp4/BoxParser.java#L627
timestamps[i] = timestampTimeUnits + timestampOffset;  // ctts == null → always 0 → PTS = DTS

That definite-but-wrong PTS is exactly what shouldDropOutputBuffer (MediaCodecVideoRenderer.java:2166) later measures against the audio clock and drops.

So: not a MediaTek defect, not an ExoPlayer bug in the usual sense — one demuxer treats a missing ctts as "unknown," the other treats it as "zero," and that's the entire difference in outcome for this class of file.

Next: I'm going to look at overhauling this to give media3 a first-class notion of "missing PTS," and only fall back to today's spray-and-pray frame-dropping behavior in that specific case.

ok - i think this approach is way more contained and correct. @FongMi please let me know what you think!

This is much better scoped than the previous version, but I found several correctness issues that should be addressed before this is ready:

1. hasReliablePresentationTimestamps is only set near the final return path of BoxParser.parseStbl(). The no-edit-list, omitTrackSampleTable, zero-duration-edit, and other early-return paths bypass it. As a result, many video tracks without ctts will never activate the workaround.

2. Missing ctts alone is too broad a trigger. A correctly muxed video without frame reordering legitimately does not need ctts. Media3 already parses reordering information such as Format.maxNumReorderSamples; the fallback should also require evidence that the bitstream actually reorders frames.

3. Returning false unconditionally from both shouldDropOutputBuffer() and shouldDropBuffersToKeyframe() disables recovery from genuine decoder slowness. On an overloaded device, video may remain behind audio indefinitely.

4. The FIFO is gated using the current inputFormat when queuing and dequeuing, while the drop logic checks codecInputFormat. During seamless format changes these may describe different buffers, leaving stale FIFO entries or applying the wrong policy.

5. Tunneling needs explicit handling. In tunneling mode, decoded buffers are not returned to the renderer, so arrivalOrderPtsQueue will be filled but never drained, and the relabeling cannot take effect.

6. The VFR test without B-frames does not validate the difficult case. With no reordering, input and output order already match. A VFR stream with B-frame reordering and missing composition timestamps cannot have its original presentation timing reconstructed exactly.

Please also add focused tests covering:

• no ctts and no edit list;
• legitimate no-ctts, non-reordered video;
• reordered video without ctts;
• flush, seek, EOS, and seamless format changes;
• decoder input/output count mismatch;
• tunneling behavior.

The investigation and the move away from a synthetic CFR timeline are solid improvements, but the current implementation still has correctness and lifecycle gaps.

Addressed point 2 in ce9dde0: the trigger now also requires track.format.maxNumReorderSamples != 0 (from the bitstream's own SPS, already parsed before this point), so a correctly-muxed non-reordering track no longer gets flagged just for lacking ctts — verified on-device that good_95s.mp4 (maxNumReorderSamples=0) now correctly skips the workaround while broken_95s.mp4 (maxNumReorderSamples=2) still triggers it. Added a synthetic MPEG-4 Part 2 test file exercising the conservative NO_VALUE case (a codec media3 never derives maxNumReorderSamples for at all) to the repro repo, with the exact generation steps: https://github.com/2bitoperations/google-tv-streamer-hevc-decoder-repro#second-file-mp4v_no_ctts_no_maxreordermp4

Thanks for the detailed review, and sorry for the long gap. I've reworked this against all six
points and the tests you asked for. I've also rebased onto main, since CONTRIBUTING asks for
PRs against main rather than release. Point by point:

1. Flag not set on early-return paths. Fixed, though the placement in my first attempt
(4a4132e, top of parseStbl) was wrong in a way worth calling out. FragmentedMp4Extractor
also calls parseTraks → parseStbl, on the moov's empty sample table, which never has a
ctts because fragmented MP4 carries composition offsets in trun. So that version flagged
essentially every fragmented B-frame video track (DASH, HLS-fMP4) as unreliable. The check now
runs immediately after the sampleCount == 0 return. It covers every path that has samples
(including no edit list, FLAG_OMIT_TRACK_SAMPLE_TABLE and edit-list paths), and it never applies
to fragmented MP4. A regression test covers this.

2. Missing ctts alone is too broad. Done. The trigger also requires
maxNumReorderSamples != 0, from the SPS. NO_VALUE is treated as "may reorder".

3. Don't disable recovery from genuine slowness. Agreed, and I removed the drop-policy change
entirely. MediaCodecVideoRenderer is now identical to main. With the relabeling below, a
constant-frame-rate track gets exact display-order timestamps, so earlyUs is a real lateness
signal again and the stock thresholds apply unchanged. On device, the affected file plays with 0
dropped frames under the stock policy, including across seeks (table below).

4. inputFormat vs codecInputFormat during format changes. The output side no longer depends
on format bookkeeping at all. Each sample is recorded at queue time (gated on codecInputFormat,
the format of the codec it's being queued to), together with the timestamp actually passed to the
codec. Each dequeued buffer is matched against those recorded timestamps by the value the codec
echoes. A buffer whose echo wasn't recorded (e.g. a reliable-timestamp format after a seamless
switch) keeps the codec's timestamp; a recorded one is relabeled. Tests cover a full codec
reinitialization mid-drain and a seamless switch.

5. Tunneling. Nothing is recorded in tunneling mode, since output buffers are never dequeued
there. A test checks that the queue stays empty.

6. VFR + reordering can't be reconstructed. Agreed, and the old test didn't exercise it. With
no ctts and reordered frames, the true presentation instant of each frame isn't in the file.
The fallback replays the queued (decode-time) timestamps in output order, which gives a
monotonic timeline that preserves the stream's sample spacing but not which frame each gap
belongs to. The test now uses real reordering with irregular spacing and asserts exactly that
behavior, and the Format javadoc says the timestamps aren't ground truth.

Input/output count mismatch (from your test list, and the most significant thing this round
found). The arrival-order queue assumes one output per recorded input, and two real cases break
that:

• With BUFFER_FLAG_DECODE_ONLY (API 34+), the codec doesn't output the buffer. Such samples are
  no longer recorded.

• After seeking to a CRA picture, decoders discard the associated RASL pictures without output.
  That's x265's default open-GOP structure. Previously this left stale entries, and every later
  frame was relabeled a few frames early for the rest of the session.

  Now, when an output's echoed timestamp is found among the pending entries, any entries queued
  more than maxNumReorderSamples positions before it can't legally still be awaiting output
  (that's the bound sps_max_num_reorder_pics defines), so they're dropped and the queue is back
  in step. A hard cap bounds the queue against a codec that never echoes exact values.

Tests (your list → where):

• no ctts, no edit list / with edit list / FLAG_OMIT_TRACK_SAMPLE_TABLE →
  Mp4ExtractorNonParameterizedTest (three small generated H.264 assets)
• legitimate no-ctts, non-reordered → Mp4ExtractorNonParameterizedTest
• reordered without ctts → extractor tests above, plus
  MediaCodecVideoRendererTest#render_withUnreliablePtsAndReorderedOutput_relabelsOutputInArrivalOrder
• ctts present / fragmented MP4 → stays reliable (extractor tests)
• flush/seek → MediaCodecRendererTest#resetPosition_withUnreliablePts_clearsArrivalOrderPtsQueue
• EOS → MediaCodecRendererTest#render_withUnreliablePtsUntilEnded_drainsArrivalOrderPtsQueue
• seamless format change →
  MediaCodecVideoRendererTest#render_withSeamlessChangeToReliablePts_onlyRelabelsSamplesOfUnreliableFormat;
  full codec reinit → MediaCodecVideoRendererTest#render_withIncompatibleFrameRateChangeUpToSdk29_discardsCodec
• VFR + reordering → MediaCodecVideoRendererTest#render_withUnreliablePtsAndVariableFrameSpacing_relabelsWithQueuedTimestamps
• input/output count mismatch →
  MediaCodecVideoRendererTest#render_withUnreliablePtsAndSamplesDiscardedByCodec_resynchronizesRelabeling
• tunneling → MediaCodecRendererTest#feedInputBuffer_withUnreliablePtsAndTunneling_doesNotGrowArrivalOrderPtsQueue

On-device (Google TV Streamer 4K, c2.mtk.hevc.decoder), demo app built from this branch:
stock drop policy, each file 95 s of 1916x816 HEVC at 24 fps (2280 samples), played to the end:

In the seek run, the prune fired once, on the first frame after seeking to a CRA, when the
decoder discarded its RASL pictures. Relabeling stayed in step from there to the end.

Unit tests: extractor mp4 package 1628/1628, MediaCodecRendererTest and
MediaCodecVideoRendererTest all passing (5 existing AV1 skips).